Transmarking of multimedia signals

ABSTRACT

The presently claimed invention relates generally to digital watermarking and data hiding. One claim recites a method including: receiving data representing video, the data comprising at least first digital watermarking embedded therein; decoding the first digital watermarking embedded in the data to obtain a first identifier; and then converting the data into low bandwidth Internet video; and embedding at least the first identifier with digital watermarking into converted low bandwidth Internet video, the identifier identifying at least the video. Of course, other combinations are described, enabled and claimed as well.

RELATED APPLICATION DATA

This application is a continuation of U.S. patent application Ser. No.09/810,080, filed Mar. 16, 2001 (U.S. Pat. No. 7,373,513), which claimsthe benefit of U.S. Provisional Patent Application No. 60/190,481, filedMar. 18, 2000.

The Ser. No. 09/810,080 application is a continuation-in-part of U.S.patent application Ser. No. 09/404,292, filed Sep. 23, 1999 (U.S. Pat.No. 7,197,156), which claims the benefit of U.S. Provisional ApplicationNos. 60/101,851, filed Sep. 25, 1998, and 60/110,683, filed Dec. 2,1998.

The Ser. No. 09/810,080 application is also a continuation-in-part ofU.S. patent application Ser. No. 09/563,664, filed May 2, 2000 (U.S.Pat. No. 6,505,160).

Each of the above patent documents is hereby incorporated by reference.

TECHNICAL FIELD

The invention relates to multimedia signal processing, for example,steganography, digital watermarking and data hiding.

BACKGROUND AND SUMMARY

Digital watermarking is a process for modifying physical or electronicmedia to embed a machine-readable code into the media. The media may bemodified such that the embedded code is imperceptible or nearlyimperceptible to the user, yet may be detected through an automateddetection process. Most commonly, digital watermarking is applied tomedia signals such as images, audio signals, and video signals. However,it may also be applied to other types of media objects, includingdocuments (e.g., through line, word or character shifting), software,multi-dimensional graphics models, and surface textures of objects.

Digital watermarking systems typically have two primary components: anencoder that embeds the watermark in a host media signal, and a decoderthat detects and reads the embedded watermark from a signal suspected ofcontaining a watermark (a suspect signal). The encoder embeds awatermark by altering the host media signal. The reading componentanalyzes a suspect signal to detect whether a watermark is present. Inapplications where the watermark encodes information, the readerextracts this information from the detected watermark.

Several particular watermarking techniques have been developed, and, forrobust watermarks, the goal is to design an imperceptible watermark thatsurvives transformation. However, this cannot always be accomplished.The reader is presumed to be familiar with the literature in this field.Particular techniques for embedding and detecting imperceptiblewatermarks in media signals are detailed in the assignee's co-pendingapplication Ser. No. 09/503,881 (U.S. Pat. No. 6,614,914) and U.S. Pat.No. 5,862,260, which are hereby incorporated by reference.

The invention provides methods and related systems, devices and softwarefor transmarking media signals. Transmarking relates to convertingauxiliary data embedded in a media signal from one digital watermarkformat to another. It is used in processes that transform the mediasignal, such as compression, broadcast, editing, rendering, etc., tochange the characteristics of the embedded watermark so that thewatermark has improved robustness or perceptibility characteristics forits new environment. In some cases, transmarking can be extended tocases where out-of-band data file the header or footer of a media file,or other metadata provided with the media file is transmarked into awatermark or is derived from a watermark. Thus, the watermarks appear tobe robust to all transformations.

One aspect of the invention is a method of transmarking a media signalpreviously embedded with a first digital watermark using a first digitalwatermark embedding method. This transmarking method detects the firstdigital watermark in the media signal. It then embeds messageinformation from the first digital watermark into a second digitalwatermark in the media signal before the media signal undergoes atransformation process. The second digital watermark is adapted tosurvive the transformation process.

Another aspect of the invention is another method of transmarking amedia signal. This method detects the first digital watermark in themedia signal, converts the media signal to a different format, andembeds message information from the first digital watermark into asecond digital watermark in the converted media signal. The seconddigital watermark is adapted to robustness or perceptibility parametersassociated with the new format.

Further features will become apparent with reference to the followingdetailed description and accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating a transmarking process where a firstdigital watermark in a media signal is transmarked into a second digitalwatermark in the media signal.

DETAILED DESCRIPTION

In many applications, a digital watermark signal embedded in mediasignals like audio, video and still images can be changed when thesignal is transformed. Transmarking of the digital watermark may be usedto change the embedded digital watermark technique at signaltransformation points to be compatible with the new signal.

For example, when playing DVD audio over the radio, analog or digitalradio, the watermark can be retrieved and re-embedded at a higher levelor using a different technique at the broadcast location. Additionally,the watermark could be modified at a repeater station due to theincreased noise level in the signal. This way an audio application canretrieve the watermark, while the original DVD can have the lowestchange in perception due to the watermark as possible. Morespecifically, the audio application may be retrieving the watermark in anoisy room and artist won't complain that the DVD watermark ruins theirrecording.

This is a continuation of the subject matter in US Patent Application,Ser. No. 09/404,292, filed Sep. 23, 1999 (U.S. Pat. No. 7,197,156),which was based upon Provisional Applications Ser. Nos. 60/101,851 and60/110,683 filed Sep. 25, 1998 and Dec. 2, 1998, respectively, allincluded herein by reference. These patent applications discussedchanging the watermark type when audio was converted from raw PCM formatto a compressed, such as MP3, AAC, Real, Liquid or other similar format.

This method also applies to video signals. For example, when watermarkedDVD video is transferred to low bandwidth Internet video, such asprovided by Real Networks, the DVD watermark is read and increased inamplitude or re-embedded to survive the massive compression needed tostream video over low bandwidth. This watermark may be used for copyprotection, but could also be used to enable links or information aboutthe video.

In some applications, it may be useful to convert auxiliary informationembedded in a media signal from one format to another. This convertingprocess is another application of transmarking. Transmarking may includeconverting an out of band identifier like a tag in a header/footer to awatermark or vice versa. It may also involve converting a message in onewatermark format to another. The process involves a decoding operatingon an input media object, and an encoding of the decoded informationinto the media object. It may also involve a process for removing themark originally in the input object to avoid interference with the newlyinserted mark.

There are a variety of reasons to perform transmarking. One is to makethe embedded information more robust to the types of processing that themedia object is likely to encounter, such as converting from onewatermark used in packaged media to another watermark used incompressed, and electronically distributed media, or a watermark used inradio or wireless phone broadcast transmission applications.

This type of transmarking process may be performed at various stages ofa media object's distribution path. An identifier in a watermark or fileheader/footer may be encoded at the time of packaging the content fordistribution, either in an electronic distribution format or a physicalpackaged medium, such as an optical disk or magnetic memory device. Atsome point, the media signal may be converted from one format toanother. This format conversion stage is an opportunity to performtransmarking that is tailored for the new format in terms of robustnessand perceptibility concerns. The new format may be a broadcast formatsuch as digital radio broadcast, or AM or FM radio broadcast. In thiscase, the identifier may be transmarked into a watermark or othermetadata format that is robust for broadcast applications. The newformat may be a compressed file format (e.g., ripping from an opticaldisk to an MP3 format). In this case, the identifier may be transmarkedinto a file header/footer or watermark format that is robust andcompatible with the compressed file format.

The transmarking process may leave an existing embedded identifier intact and layer an additional identifier into the media object. This mayinclude encoding a new watermark that does not interfere with anexisting watermark (e.g., insert the new watermark in unmarked portionsof the media object or in a non-interfering transform domain). It mayalso include adding additional or new identifier tags to headers orfooters in the file format.

FIG. 1 is a flow diagram illustrating a process of transmarking. Theinput to the transmarking process is a digitally watermarked signal 20,such as an audio signal (e.g., a music track), a video signal, or stillimage. The digital watermark carries a message payload of one or moresymbols (e.g., binary or M-ary symbols) conveying information such as acontent identifier, transaction identifier, database index, usage orcopy control parameters (flags instructing a device or process not tocopy, copy once, not to transfer, etc.). There are a variety ofapplications for digital watermarks in multimedia content, includingforensic tracking, broadcast monitoring, copy control, and using thewatermark as a trigger for or link to interactive content to be renderedalong with the watermarked signal, either in response to user input orautomatically as the watermarked signal is playing. Some of theseapplications are discussed in co-pending patent application Ser. No.09/571,422 (U.S. Pat. No. 6,947,571), Ser. No. 09/563,664 (U.S. Pat. No.6,505,160), Ser. Nos. 09/574,726, and 09/597,209 (U.S. Pat. No.6,411,725) which are hereby incorporated by reference.

In these applications, there are a number of reasons to transmark thewatermark signal embedded in a host signal. Some examples include: toincrease the robustness of the watermark as it undergoes a format change(such as for compression, transmission, digital to analog conversion,up-sampling or down-sampling, printing, display, etc.), to reduce theperceptibility of the watermark before playback, or to balance thetrade-off of perceptibility levels vs. robustness levels of thewatermark signal for a new as the host signal undergoes a change fromone format to another.

The transmarking process illustrated in FIG. 1 begins by detecting afirst watermark in the watermarked signal (22). A watermark detectoremploys a watermark key to identify the presence of a watermark. Thespecific operation of the detector depends on the watermarking processemployed. In many techniques, the watermark key specifies the spatial,time, and/or frequency domain location of the watermark signal. It mayalso specify how to decode a message that has been modulated with apseudo-random number (e.g., frequency or phase hopping, spread spectrummodulation). To simplify the search for the watermark, the watermarkdetector searches for reference signal attributes of the embeddedsignal, such as a known sequence of embedded symbols, or a known signalpattern in a particular time, space, or transform domain. Theseattributes enable the detector to determine whether a watermark ispresent in a suspect signal, and to determine its position within thetime, space and/or transform domain.

Next, the watermark detector may optionally decode an embedded message(26), such as copy control parameters, content identifiers, owneridentifiers, transaction identifiers, etc. This step is optional becausethe initial detection operation may convey enough information to triggerthe remainder of the transmarking operation. For example, the meredetection of the presence of a watermark signal at a particular time,space, or transform domain location may convey one or more bits ofmessage information.

Some examples will help illustrate the detection and message decodingprocess. One type of watermark embedding process encodes symbols byinserting scaled-amplitude, shifted versions of the host signal. Theshift may be a combination of time, frequency, and/or spatial shifts ofthe host signal depending on the nature of the signal (e.g.,time-frequency for audio, spatial frequency for imagery). This shiftedversion conveys message symbol values by the presence or absence of theshifted version or versions at a particular shift relative to the host,and/or by the amount of change effected to a statistical characteristicof the host signal by the embedding of the shifted version. Another typeof embedding process embeds a watermark by modulating perceptual domainsamples and/or transform domain frequency coefficients. In both cases,the message may be randomized by applying a pseudo randomizing process(e.g., spreading a message by multiplying or XORing with a PN sequence)before making the changes to the host to hide the resulting messagesequence in the host signal. The message may be embedded by an additiveprocess of a modifying signal and/or by a quantization of sample values,frequency coefficient values, or statistical characteristic values.

In these embedding techniques, the detector looks for attributes of thewatermark signal, such as by using correlation or a statistical analysisto detect the shifted versions or modulated samples/coefficients. Byidentifying evidence of known symbols or watermark signal attributes,the detector determines whether a watermark signal is present. In somecases, the watermark detector determines that an additional messagepayload message is present based on the detection of certain watermarksignal attributes. It then proceeds to decode additional signalattributes and map them to message symbols. Further error correctiondecoding may be employed, such as BCH, turbo, Reed Solomon, andconvolution decoding, to extract the message payload.

Next, the transmarking process removes the first watermark signal (28).Again, this process is optional because the transmarking process mayproceed by embedding a second watermark without specifically attemptingto remove or mitigate the effects of the first. Once the watermarkdetector has detected the watermark and determined its temporal,spatial, and/or frequency domain position, it can remove the watermarkor mitigate its effect. It can substantially remove the watermark incases where the embedding function is invertable, such as a reversibleaddition operation, by performing the inverse of the embedding functionusing the watermarking key to specify the attributes and location of thefirst watermark. It can also remove the watermark without knowing theinverse function, such as using a whitening filter with PN sequencebased watermarking.

Interestingly, this could allow a less perceptible watermark to be addedto content that is going from a low quality medium to a higher qualitymedium. Although the content will still be the quality of the originalmedium, the watermark will produce minimal or no further qualitydegradation. When transforming from high quality to lower qualitymedium, removing the first watermark still improves quality androbustness due to reducing interference between each watermark.

In some applications, the watermarked signal may be converted to anotherformat, such as compressing the signal before the transmarking processproceeds. These applications are ones where the signal in the new formatis available for watermarking. In this case, the transmarking processproceeds by embedding a second watermark into the host signal after theformat change has occurred. This enables the watermark embedding processto adapt the watermark to the perceptual quality and robustnessparameters of the signal in the new format. In other applications, suchas where the signal is broadcast, it is difficult or not practicallypossible to intercept the signal for embedding a new watermark after theformat change occurs. For example, the format change may occur as aresult of the broadcast transmission. In this case, the transmarkingprocess proceeds to embed a second watermark and adapts the watermark tothe robustness and perceptual quality parameters appropriate for the newformat of the signal before the format change occurs.

Next, the transmarking process encodes the second watermark (44) usingthe same or some different embedding process as the first watermark(30). This second watermark can be added before the transformation,after the transformation, or during the transformation with a feedbackloop. For example, the first watermark may be embedded by adding ashifted version of the host signal, while the second watermark may beembedded by adding a perceptually adapted pseudo random carrier signalin the perceptual or some transform domain (like Fourier, DCT, wavelet,etc.), or vice versa. The second watermark may modify differenttemporal, spatial or frequency portions of the host signal than thefirst, or the two watermarks may overlap in one or more of theseportions of the signal. Regardless of whether the watermark embeddingfunction is fundamentally the same or different as the one used to embedthe first watermark, this embedding process (30) is specifically adaptedto the perceptibility and robustness constraints of the new format orenvironment. This watermark embedding process uses robustness parameters(32) (e.g., watermark signal gain, extent of redundancy, frequencydomain locations) to specify the watermark strength, redundancy andfrequency domain locations that particularly adapt the watermark forsurvival in the new format. This second watermark may add newinformation about the transformation where this information can be usedfor forensic tracking. The information could include any combination ofthe following: an identifier of the transformation device (such as anMPEG encoder device or manufacturer), and an identification of thedistribution system, such as an identifier of the broadcast network orcable system. This new information augments the original informationembedded into the first watermark and does not alter its meaning, butinstead, adds additional payload information.

To ensure that the second watermark satisfies robustness constraints,the embedding process optionally applies a feedback path that appliesthe watermarked signal to a degradation process, then measures thenumber of errors incurred in decoding a known message, and selectivelyincreases the gain of the watermark signal in the portions (temporal,spatial or frequency portions) of the signal where the errors occurred.The degradation operations may include a compression operation, or anoperation that models degradation likely to be encountered in the newformat, such as digital to analog conversion, printing/scanning,broadcast transmission, time scale changes, etc. This process repeatsuntil the measured error rate falls below an acceptable threshold.

In addition, the embedding process uses perceptual quality parameters 33that specify constraints on perceptual quality of the signal for the newformat. These parameters may specify limits on the watermark strength,or define a perceptibility threshold that can be measured automatically,like Peak Signal to Noise Ratio, typically used in analysis of digitalwatermarking methods. Again, as above, the embedding process optionallyincludes a feedback path that measures the perceptual quality of thewatermarked signal and selectively reduces the gain of the signal in theportions of the signal (temporal, spatial or frequency portions) wherethe watermarked signal exceeds the perceptibility threshold.

FIG. 1 graphically depicts the interaction between the watermarkembedding process 30, on the one hand, and the rendering/editingenvironment or transmission environments (34, 36) on the other. Thisdiagram depicts how the embedder adapts the new watermark to theenvironment in which the transmarked signal will be used. For example,if the signal is a still image that is being used in a photo editingsoftware environment, the robustness of the watermark can be adapted tothe image processing operations in the editing tool. If the watermark isgoing to need to survive printing, then the transmarking process embedsthe signal with a new watermark designed to survive that process and berecoverable via an image scanned from the printed image. In this case,the watermark embedder may include additional calibration signalinformation as set forth in U.S. Pat. No. 5,862,260 to ensure that thewatermark can be detected despite geometric distortion.

As an aside, just as the second watermark may be adapted to the intendedenvironment, the operations in the editing tool can be modified so as toimprove the survivability of the watermark. In this case, the imageediting operations such as blurring, color transformation, etc. areadapted to preserve the watermark signal to the extent possible. Inparticular, a low pass filter or blur operation that typically reduceshigh frequency components may be implemented so as to pass selected highfrequency components to maintain the watermark signal in thosecomponents. The operation of adding guassian noise may be modified byshaping or reducing the noise at certain frequencies to reduceinterference with the watermark signal at those frequencies. In caseswhere watermarks are inserted by modifying a particular color channelsuch as luminance, the color transform operations may be designed topreserve the luminance of the watermarked image.

Further, the signal editing tool may be integrated with the transmarkingprocess to decode the watermark before an operation, and then re-encodethe watermark after an operation to ensure that it is preserved. Forexample, the wateramark may be re-applied after the image editing toolis used to make an affine transform of an image, or after the image iscropped.

In the case of transmission of media signals over a communicationchannel, the watermark may be transmarked at points in the communicationchannel where the signal (audio, video, or image signal) is transformed.These include cases where the signal is un-compressed and re-compressedin another format, where the signal is transformed in a router orrepeater (e.g., when the signal is amplified in a router or repeaternode in the communication path, the watermark is transmarked at higherintensity), where the signal is transformed into packets in a switchingnetwork, the watermark signal may be decoded and re-encoded in theindividual packets, or re-encoded after the signal is re-combined. There-encoding is effected by transferring a watermarking command in theheader of the packets specifying the watermark payload and watermarkembedding protocol to be used in the re-combined signal.

In audio and video compression codecs, the transmarking process may beintegrated into the compression codec. This enables the codec to modifythe compression operation or modify the bitrate to ensure that thewatermark survives. In the first case, the compression codec may bedesigned to preserve certain frequency components that would otherwisebe substantially reduced to preserve the watermark. In the latter case,the codec selects a bit rate at which the watermark survives, yet thesignal has been compressed to an acceptable level.

If the watermarked signal is going to be rendered in a high fidelitydevice where usage is tightly controlled, such as a DVD player, thesecond watermark can be embedded so as to have less impact onperceptibility. Conversely, if the watermarked signal is going to berendered in a lower fidelity device, such as a personal computer, thesecond watermark can be embedded so that it is more robust while stayingwithin the perceptual quality parameters of the rendering device. Inaddition, the watermark can be changed if DVD audio masters areconverted to CDs or cassette tapes.

If the watermarked signal is going to be transmitted, such as in thebroadcast environment, the embedding process encodes the secondwatermark with robustness to survive the broadcast and maintain theperceptual fidelity within the less rigid constraints of the broadcastenvironment. The transmarking process can be used to encode triggersused in interactive video or audio. The triggers may be originallyencoded in one format and transmarked into another format beforebroadcast, or at some node in the broadcast process. For example, thetrigger can transmarked in video when it is compressed into MPEG2 formatfor broadcast, or when the content is received at a cable head-end ornode in the content distribution channel. The trigger may be a networkaddress of interactive content like an IP address or URL, or an index toa network address, interactive content like HTML or XML, etc.

As another example, triggers for interactive content in radio broadcastscan be transmarked when the content is transferred from a packagedmedium, such as an optical disk, and prepared for broadcast overtraditional radio broadcast, digital satellite broadcast, or Internetstreaming broadcast.

Like the first watermark, this second watermark employs a watermarkingkey 38 to specify the spatial, time and or frequency attributes of thesecond watermark. In addition, the message decoded from the firstwatermark, such as an identifiers 40, copy control parameters 42 areembedded.

The result of the transmarking process, in a typical case, is a newwatermarked signal 46. As noted, the information or function of thewatermark may be transmarked to out-of-band data like a file header orfooter, such as an ID3 tag in MP3 audio. Conversely, out-of-band datamay be transmarked into in-band data that is embedded into the hostsignal using a digital watermarking process.

CONCLUDING REMARKS

Having described and illustrated the principles of the technology withreference to specific implementations, it will be recognized that thetechnology can be implemented in many other, different, forms. Toprovide a comprehensive disclosure without unduly lengthening thespecification, applicants incorporate by reference the patents andpatent applications referenced above.

The methods, processes, and systems described above may be implementedin hardware, software or a combination of hardware and software. Forexample, the auxiliary data encoding processes may be implemented in aprogrammable computer or a special purpose digital circuit. Similarly,auxiliary data decoding may be implemented in software, firmware,hardware, or combinations of software, firmware and hardware. Themethods and processes described above may be implemented in programsexecuted from a system's memory (a computer readable medium, such as anelectronic, optical or magnetic storage device).

The particular combinations of elements and features in theabove-detailed embodiments are exemplary only; the interchanging andsubstitution of these teachings with other teachings in this and theincorporated-by-reference patents/applications are also contemplated.

What is claimed is:
 1. A method comprising: receiving data representingvideo, the data comprising at least first digital watermarking embeddedtherein; using a processor, decoding the first digital watermarkingembedded in the data to obtain a first identifier in which said decodingutilizes a watermark decoding key; and then converting the data into lowbandwidth Internet video; using the processor, embedding at least thefirst identifier, obtained from said act of decoding the first digitalwatermarking embedded in the data, with second digital watermarking intoconverted low bandwidth Internet video, the first identifier identifyingat least the video, in which said embedding utilizes a watermarkingembedding key and robustness parameters.
 2. The method of claim 1further comprising embedding a second identifier in the converted lowbandwidth Internet video data with third digital watermarking.
 3. Themethod of claim 2 wherein the second identifier is associated with atransaction.
 4. The method of claim 2 wherein the second identifier isassociated with a data conversion.
 5. An apparatus comprising:electronic memory for storing data representing video, the datacomprising at least first digital watermarking embedded therein; aprocessor programmed for: utilizing a watermarking decoding key,decoding the first digital watermarking embedded in the data to obtain afirst identifier; and then converting the data into low bandwidthInternet video; utilizing a watermarking embedding key and robustnessparameters, embedding at least the first identifier, obtained from saiddecoding the first digital watermarking embedded in the data, withsecond digital watermarking into converted low bandwidth Internet video,the identifier identifying at least the video.
 6. The apparatus of claim5 in which the processor is further programmed for embedding a secondidentifier in the converted low band width Internet video data withthird digital watermarking.
 7. The apparatus claim 6 wherein the secondidentifier is associated with a transaction.
 8. The apparatus of claim 6wherein the second identifier is associated with a data conversion.
 9. Anon-transitory computer readable medium comprising instructions storedthereon that when executed cause a processor to perform the following:controlling data representing video, the data comprising at least firstdigital watermarking embedded therein; utilizing a watermarking decodingkey, decoding the first digital watermarking embedded in the data toobtain a first identifier; and then converting the data into lowbandwidth Internet video; utilizing a watermarking embedding key androbustness parameters, embedding at least the first identifier, obtainedfrom said act of decoding the first digital watermarking embedded in thedata, with second digital watermarking into converted low bandwidthInternet video, the first identifier identifying at least the video. 10.The non-transitory computer readable medium of claim 9 in which theinstructions that when executed further cause the processor to preform:embedding a second identifier in the converted low bandwidth Internetvideo data with third digital watermarking.
 11. The non-transitorycomputer readable medium of claim 10 in which the second identifier isassociated with a transaction.
 12. The non-transitory computer readablemedium of claim 10 in which the second identifier is associated with adata conversion.